Machine-vision-based obstacle detection and tracking is an important component in autonomous vehicle systems. In a typical driving scene, the obstacles include vehicles, pedestrians and any other objects that are either moving or rising above the road plane. The purpose of obstacle detection is to separate moving objects from the driving scene, the obstacles including vehicles, pedestrians and any other objects that are either moving or rising above the road plane. Such information is required by a number of automotive applications, e.g., adaptive cruise control, forward collision avoidance and lane departure warning. By fusing the results of detecting and tracking individual objects, it is possible to achieve sufficient perception of the driving environment.
A number of approaches have been developed to address vehicle tracking and detection. One category of obstacle detection methods uses motion analysis. The true motion vectors of the road plane resulting from camera motion are estimated from camera parameters, position and ego velocity. When an object is violating the planarity assumption, a significant difference between the optical flow vectors calculated from consecutive frames and the true motion vectors (i.e., motion parallax) is detected, which indicates the appearance of an obstacle that is either moving or rising above the road plane. Other approaches group the optical flow vectors into clusters and track with a Kalman filter. With a calibrated camera and estimate of ego-motion, one approach can detect and distinguish between moving objects and stationary but elevated objects. In yet another approach, the clusters of flow vectors are further grouped into parallel lines on the rear face of an enclosing box, which correspond to the rear bumper of moving vehicles on the road plane.
Ego-motion recovery is also used to track vehicle position. Planar motion models are used to describe the 2D image flow of road plane resulting from camera motion. Image measurements that violate the rigid motion assumption and structures above the road plane are considered outlier and excluded from ego-motion estimation. In yet another approach, the motion analysis is extended to automatically detect multiple planes from sparse optical flow information.
Obstacle detection methods using only motion information are sensitive to image noise and illumination changes. In addition, the detected obstacles cannot be classified by motion information alone.
In another approach, a 3D polyhedral model is used to detect and track vehicles in a surveillance setup. A vehicle is described in a 2D view sketch composed of edge segments specified by their length parameters. This approach requires previous knowledge of ego-motion and camera calibration. When the camera parameters and ego-vehicle velocity are not available, motion analysis is no longer sufficient for vehicle detection because there is no way of knowing the true image flow describing the road plane.
In another known approach, distant cars are detected and validated through horizontal and vertical edge detection, aspect ratio check and template correlation. If camera parameters are available, the distance of the detected cars from ego-vehicle can be estimated through the perspective projection model. In some instances, the detection starts from feature selection and tracking. Tracked features are grouped into clusters corresponding roughly to different objects in the scene. Vehicles are identified and validated through edge finding. A simple vehicle classification scheme is introduced based on the aspect ratio to distinguish cars, SUVs, pickups and trucks. In other approaches, a probabilistic model is introduced to model the strength of the edges around the vehicle boundary. Vehicle detection is implemented by locating bounding boxes from edges and verifying the vehicle presence with the edge model. The extended Kalman filter is used to track the detected vehicle.
A second category of vehicle detection algorithms treats the detection as a two-class pattern classification problem involving the vehicle class and the non-vehicle class. Instead of using the empirical descriptions for the vehicle class, these algorithms use a classification function to tell if an image patch contains a vehicle or not. Through an offline training process, the best classification function with minimum classification error is learned from a number of vehicle and non-vehicle examples. The training process takes into account the variation of vehicle appearance within the training examples. Compared to the empirical vehicle model with edges, shapes and templates, the training and classification approach produces more reliable detection results.
In one approach, a vehicle detection algorithm is implemented in two steps: multi-scale driven hypothesis generation and appearance-based hypothesis verification. Appearance-based hypothesis verification verifies the hypothesis using wavelet feature extraction approach and Support Vector Machines (SVMs) as classifiers. In another approach, a Support Vector Tracking (SVT) method is introduced, which integrates the SVM classifier into an optical-flow based tracker. Instead of minimizing an intensity difference function between consecutive frames, SVT maximizes the SVM score such that the detection results have the highest confidence scores in the corresponding video frames.
In these approaches, the classifier response is computed over a neighborhood region around the vehicle position detected from the previous frame. The location with the highest response is considered the vehicle position in the current frame. The focus of such detection methods is to build an accurate and efficient vehicle detector (classifier). Tracking is considered a process of data association that links the detection results from individual frames to a temporal trajectory.
Even though all of these different approaches exist, there is a need for a method for detecting and tracking vehicles that has high performance and low complexity. Since the calculations required for vehicle detection and tracking must be made in real time, complex calculations are not feasible. Yet, in order to acceptably track a vehicle, the accuracy of the detection method must be high.